DA1074T9 


J. 


UNCLASSIFIED' 
UNLIMITED  DISTRIBUTION 


CRDV  RAPPORT  4198  81 
DOSSIER  3621 J  012 
AVRIL  1981 


APRIL  1 98 1 


1M\0I  >1  T.M!-.\  |  \TH>\  \\  I'l'll  (  T  >i  n|{  \M)  r I  \  I  I  i{ | 


J.  i' .  Hun  Iter 


Centre  de  Recherches  pour  la  Defense 
Defence  Research  Establishment 


IjffiMBUtlON  ~STA’rCgyy~  A 

Approved  for  public  re loc* o; 
DWrbutlon  liclhujted 


i _ _ 


Valcartier,  Quebec 


BUREAU  RECHERCHE  El  DEVEEOPPEMENT 
MINISTERS  OE  LA  DEFENSE  NATIONALS 
CANADA 


1)1 


Non  Li  ASS  I  f  :• 
I-  I  :  >,\  H  , J  V 


RESEARCH  AND  DEVELOPMENT  BRANCH 
DEPARTMENT  OF  NATIONAL  DEFENCE 
CANADA 


81  11  09  069 


CRDV  R -4198/81 
DOSSIER:  3621J-012 


UNCLASSIFIED 


Quebec, 


DREV  —  R-4198/81 
FILt:  76217-012 


IMAGE  SEGMENTATION  WITH  COLOR  AND  TEXTURE 


by 

I  3.  F.  Boulter 


/ 


CENTRE  DE  RECHERCHES  POUR  LA  DEFENSE 
DEFENCE  RESEARCH  ESTABLISHMENT 
VALCARTIER 
Tel.  (418)  844-4271 


Canada 


April/avril  1981 


NON  CLASSIFIE 


/ 


UNCLASSIFIED 

i 


ABSTRACT 


We  examined  how  objects  or  regions  in  an  image  formed  by  the 
reflection  of  radiation  from  a  scene  can  be  segmented  using  differences 
in  their  color  and  texture.  The  color  features  (normalized  red  content 
and  normalized  blue  content)  permit  good  spatial  resolution  of  the 
segmented  regions,  whereas  the  textural  features  (isotropic  and  linear 
textures)  help  to  distinguish  similarly  colored  objects.  The  color  and 
texture  features  were  chosen  because  of  their  low  dependence  on  viewing 
conditions,  such  as  surface  orientation  and  illumination.  As  an 
example,  we  segment  an  olive-drab  colored  military  truck  from  its 
background  of  vegetation.  (U) 


RESUME 


On  a  etudie  comment  on  peut  segmenter,  dans  une  image  produite 
par  la  reflexion  de  la  radiation  d'une  scbne,  des  objets  ou  des  regions 
en  employant  les  differences  de  couleur  et  de  texture.  Les  traits  de 
couleurs  (le  contenu  de  rouge  normalise  et  celui  de  bleu  normalise)  nous 
permettent  d'obtenir  une  bonne  resolution  spatiale  des  objets  segmentes. 
et  les  traits  de  texture  (les  textures  isotropique  et  directionnelle; 
nous  aident  h  distinguer  differents  objets  de  m&me  couleur.  Ces  traits 
sont  choisis  parce  qu'ils  sont  peu  influences  par  les  conditions 
d' observation,  telles  que  I 'orientation  ou  1 ’ illumination  des  surfaces. 
Comme  exemple,  on  segmente  un  camion  militaire  vert  de  son  arrifere-plan 
de  vegetation.  (NC) 
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1.0  INTRODUCTION 


The  objective  of  image  segmentation  is  to  classify  homogeneous 
regions  of  an  image  into  a  limited  number  of  groups.  Such  segmentation 
operations  are  becoming  increasingly  important  in  automatic  digital 
image-analysis  systems  which  are  intended  to  operate  with  little  or  no 
operator  supervision.  One  example  is  an  automatic  target-acquisition 
system  (Refs.  1-3)  that  classifies  the  regions  of  an  image  into  2 
groups:  target  and  background.  Another  is  a  remote  sensing  system  used 
for  crop  or  land-usage  classification.  In  this  document,  we  examine  the 
image  segmentation  based  on  the  combined  use  of  color  and  textural 
information,  and  we  illustrate  the  technique  by  extracting  a  military 
vehicle  from  its  background  in  a  complex  scene. 

The  image  to  be  analyzed  is  usually  the  2-dimensional  projection  of 
a  3-dimensional  scene,  and  the  segmentation  groups  may  correspond  to 
objects  that  have  similar  surface  properties.  In  some  cases,  successful 
segmentation  may  be  achieved  by  using  differences  in  gray  level  alone. 
An  example  is  the  segmentation  of  a  hot  target  from  a  cooler  background 
in  a  thermal-emission  image  (Refs.  1-3).  In  other  situations, 
segmentation  schemes  that  are  based  only  on  differences  in  gray  level 
may  fail  to  correctly  segment  a  scene  that  can  easily  be  interpreted 
visually.  For  example,  contrast  in  images  formed  by  the  reflection  of 
radiation  depends  on  the  orientation  and  the  illumination  of  the  target 
surfaces,  as  well  as  on  the  surface  material.  The  images  of  such 
targets  may  contain  gray  levels  that  are  both  higher  and  lower  than  the 
background  region  from  which  they  are  to  be  segmented.  While 
higher-order  processes,  such  as  the  analysis  of  the  overall  shape  of  the 
regions,  the  context  of  the  regions  (Refs.  A  and  5),  previous 
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experience,  etc.,  are  involved  in  the  segmentation  performed  by  the 
human  visual  system,  additional  local  features  may  also  yield  useful 
information.  In  particular,  color  (Refs.  6  and  7)  and  texture  (Ref.  8) 
are  local  features  that  may  be  easily  obtained  from  many  imaging  systems 
and  used  for  automatic  image  segmentation. 

In  Sect.  2.1,  we  review  the  general  concept  of  classifying  the 
individual  elements  of  a  digital  image  by  using  their  locations  in  an 
N-dimensional  feature  space.  The  success  of  this  procedure  depends  on 
the  choice  of  local  features  which  have  sufficiently  different  values  in 
the  regions  to  be  segmented  from  one  another.  We  want  to  minimize  the 
overlap  in  the  feature  space  between  the  "clusters"  that  correspond  to 
the  different  regions.  As  we  show  in  Sect.  2.2,  one  way  to  do  this  for 
reflection  images  is  to  use  color  ratios  (Ref.  7)  because,  ideally,  they 
contain  no  contrast  due  to  differences  in  surface  illumination  or 
orientation.  The  value  of  the  ratio  depends  only  on  the  "color"  of  the 
surface.  In  Sect.  2.3,  we  describe  2  simple  algorithms  to  measure  the 
local  amounts  of  isotropic  and  linear  textures.  In  Sect.  3.0,  we 
segment  a  target  consisting  of  a  military  truck  from  its  background  of 
vegetation  and  ground  terrain  by  selecting  the  appropriate  volume  in  the 
4-dimensional  space  spanned  by  isotropic  and  linear  textures,  and  by  red 
and  blue  color  ratios.  Spatial  averaging  of  the  color  and  texture 
features  increases  the  number  of  image  elements  correctly  classified, 
but  at  the  expense  of  a  loss  in  the  resolution  of  edge  and  other 
high-frequency  details,  and  we  examine  this  tradeoff  experimentally. 
Since  the  intention  of  the  present  document  is  to  demonstrate  the 
combined  use  of  local  color  and  texture  features  for  image 
segmentation,  we  employed  an  interactive  digital  image-processing  system 
to  locate  the  clusters  and  to  perform  the  segmentation.  Section  4.0 
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relates  these  preliminary  results  to  automatic  image  segmentation  where 
the  system  is  first  trained  with  preclassified  imagery  before  it  is 
applied  to  the  unknown  imagery. 

This  work  was  performed  at  DREV  during  the  period  June  1979  to 
December  1979  under  PCN  21312,  Target  Enhancement  Using  Spectral  and 
Textural  Information. 


2.0  THEORY 

2.1  Classification  with  Local  Features 


Suppose  that  we  want  to  classify  the  individual  elements  of  a 
digital  image  into  a  finite  number  of  groups.  These  may  correspond  to 
distinct  objects  or  surfaces  in  a  scene.  Classification  is  to  be 
achieved  by  using  only  the  local  properties  of  the  image  elements. 
Higher-order  classification,  which  may  use  macroscopic  properties  of  the 
objects  or  surfaces,  such  as  their  shape,  size,  context  in  the  scene, 
etc.,  will  be  considered  as  a  separate  operation  that  may  be  applied 
later,  after  the  individual  elements  have  been  classified. 

For  each  image  element,  we  measure  a  set  of  local  features,  i.e. 
features  that  are  either  a  property  of  the  individual  element,  or  of  the 
elements  in  the  close  neighbourhood  of  it.  The  former  may  include 
color  and  gray  level,  whereas  the  latter  may  include  texture  and  edge 
content.  If  the  average  values  of  the  features  in  the  groups  are 
sufficiently  different,  the  individual  image  elements  can  be  classified 
by  setting  appropriate  upper  and  lower  threshold  levels  on  each  feature. 


a. 
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The  first  problem  is  to  determine  which  local  features  are  the  most 
useful  for  distinguishing  the  different  groups.  For  example,  in  images 
formed  by  the  reflection  of  radiation,  we  should  use  features  that 
characterize  the  reflecting  surface  and  that  are  independent  of  viewing 
conditions,  such  as  surface  orientation  and  illumination.  The  latter 
effects  may  be  unpredictable  in  practice,  and  will  increase  the  variance 
of  a  given  feature  within  a  group.  This  increases  the  overlapping 
between  the  groups  and  makes  the  unique  classification  of  the  image 
elements  more  difficult.  The  spectral  and  textural  features  described 
in  Sect.  2.2  and  2.3  were  chosen  because  of  their  low  dependence  on  3uch 
unpredictable  effects. 

The  second  problem  consists  in  determining  the  permitted  range  of 
values  for  each  local  feature  if  that  image  element  i3  to  be  classified 
into  a  given  group.  It  is  convenient  to  view  thi3  problem  in  a 
multidimensional  space  spanned  by  the  local  features.  Ideally,  the 
different  groups  will  appear  in  well  defined  clusters  in  thi3  space,  and 
can  be  separated  by  appropriate  "decision  surfaces".  Image  elements 
will  be  reliably  classified  if  the  clusters  are  well  separated  and  do 
not  overlap  one  another.  If  clusters  that  correspond  to  different 
groups  do  overlap,  then  elements  within  the  overlap  region  may  be 
incorrectly  classified.  We  may  partially  correct  such  errors  by  using 
other  approaches,  such  as  reclassifying  image  elements  if  they  are 
surrounded  in  the  original  image  by  elements  that  have  been  classified 
into  a  different  group  (Ref.  9).  In  the  experimental  results  given  in 
Sect.  3.0,  we  use  2-dimensional  projections  to  visualize  the  target  and 
the  background  clusters  in  the  space  spanned  by  2  color-ratio  features 
and  2  textural  features,  and  we  determine  the  appropriate  decision 
surfaces  by  inspection. 
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2.2  Color  Features 


Contrast  in  an  image  formed  by  the  reflection  of  radiation  depends 
on  the  orientation  and  the  illumination  of  the  reflecting  surfaces,  as 
well  as  on  their  intrinsic  reflectivities.  To  perform  the  present 
classification,  we  wish  to  use  features  that  characterize  the  intrinsic 
surface  properties  independent  of  orientation  and  illumination. 

Suppose  we  obtain  2  registered  images  of  a  scene  in  different 
spectral  bands,  and  divide  one,  point  by  point,  by  the  other.  Under 
certain  assumptions,  the  contrast  due  to  the  differences  in  the  surface 
orientation  and  illumination  will  be  the  same  in  the  2  images,  and  will 
cancel  in  the  ratio  (Ref.  7).  Any  contrast  remaining  in  the  ratio  image 
will  be  due  to  differences  in  the  spectral  reflectivities  (i.e.  the 
"colors")  of  the  surfaces.  To  achieve  this,  we  must  assume,  first,  that 
the  illumination  of  the  scene  can  be  expressed  as  the  product  of  a 
constant,  a  part  that  depends  only  on  the  wavelength  and  another  that 
depends  only  on  the  direction  of  incidence  and,  second,  that  the 
reflectivities  of  the  surfaces  in  the  scene  can  similarly  be  expressed 
as  the  product  of  a  part  that  depends  only  on  the  wavelength  and  another 
that  depends  only  on  the  angles  of  incidence  and  reflectance.  Stated 
differently,  we  assume  that  the  way  in  which  illumination  changes  as  a 
function  of  surface  orientation  is  independent  of  wavelength,  and  so  is 
the  way  in  which  reflectivity  changes  as  a  function  of  the  angles  of 
incidence  and  reflectance. 

Spectral  ratioing  removes  information,  but  the  assumption  is  that 
the  orientation  and  the  illumination  information  is  difficult  to  predict 
or  to  allow  for  in  practice,  and  will  impede  the  correct  classification 
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of  the  image  elements.  Consider,  for  example,  an  object  that  has 
surfaces  at  different  orientations  and  which  receives  nonuniform 
illumination.  We  assume  that  all  the  surfaces  have  the  same  reflecting 
properties.  If  we  view  the  abject  through  a  red  and  a  blue  filter,  the 
2  resulting  color-component  images  will  both  contain  contrast  due  to  the 
differences  in  surface  orientation  and  illumination.  The  object  will 
occupy  an  extended  area  in  the  feature  space  spanned  by  the  red  and  the 
blue  components.  Another  similar  object,  which  is  illuminated  with 
light  with  the  same  spectral  content,  but  which  has  a  surface  color 
slightly  different  from  the  first,  will  also  occupy  an  extended  area  in 
this  feature  space.  If  the  clusters  corresponding  to  the  2  objects 
overlap  one  another,  then  the  elements  cannot  be  all  correctly 
classified  by  using  these  color  features. 

Suppose  we  construct  2  color-ratio  images  by  dividing  the  red  and 
the  blue  images  by  the  sum  of  the  two.  Both  objects  will  appear  with  a 
uniform  gray  level  in  the  2  ratio  images  (Ref.  6).  Ideally,  contrast 
due  to  differences  in  surface  orientation  and  illumination  will  be 
removed.  Each  object  will  be  restricted  to  a  single  point  in  the 
feature  space  spanned  by  the  red  and  the  blue  color  ratios,  and  elements 
belonging  to  the  2  objects  can  be  correctly  classified  by  placing  a 
decision  surface  between  the  2  points. 

2.3  Textural  Features 


Visual  texture  consists  of  repetitive  patterns  formed  by  arranging 
primitive  elements  according  to  certain  placement  rules  (Ref.  10). 
These  elements  and  rules  may  be  difficult  to  determine,  and  may  vary 
widely  from  one  type  of  surface  to  another.  As  a  result,  a  number  of 


t 


UNCLASSIFIED 

7 


other  approaches  such  as  the  measurement  of  spatial  frequency  content 
(Ref.  11)  and  counting  the  number  of  maxima  and  minima  per  unit  area 
(Ref.  12)  have  been  used  to  characterize  the  textural  properties  of 
surfaces.  We  consider  simple  primitive  elements  that  should  appear  with 
reasonably  high  probabilities  in  most  situations.  These  primitives 
correspond  to  isotropic  and  linear  textures.  Holes  in  an  acoustic  tile 
are  an  example  of  isotropic  texture,  whereas  wood  grain  is  an  example  of 
linear  texture. 

On  the  basis  of  the  3-  by  3-element  region  that  surrounds  it,  we 
assign,  to  each  element  in  the  digital  image,  one  of  three  properties •. 
isotropic  texture,  linear  texture  or  no  texture  (Ref.  8).  The  central 
element  of  that  region  is  assumed  to  possess  isotropic  texture  if  its 
qray  level  is  either  higher  or  lower  than  those  of  the  remaining  8 
surrounding  elements.  We  assign  linear  texture  to  the  central  element 
if  the  gray  levels  of  all  the  elements  along  a  line  3  elements  long  that 
passes  through  it  (the  line  can  be  either  horizontal,  vertical  or 
diagonal)  are  all  either  higher  or  lower  than  those  of  the  6  remaining 
elements.  These  algorithms  are  used  to  generate  binary  isotropic-  and 
linear-texture  images,  each  containing  a  "1"  if  that  particular  texture 
has  been  assigned  and  a  "0"  if  it  has  not. 

The  textural  features  of  most  surfaces  in  complex  scenes  will 
undergo  large  fluctuations,  even  when  they  are  measured  with  simple 
primitives.  It  is  necessary,  in  general,  to  perform  a  spatial  averaginq 
of  textural  features  before  they  can  be  used  for  classification 
purposes.  We  smoothed  the  present  binary  texture  images  with  a  low-pass 
filter  that  had  a  2-dimensional  Gaussian  point-spread  function.  This 
produced  2  continuous-tone  images  in  which  gray  level  represents  the 
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average  amount  of  isotropic  or  linear  texture  present  in  the 
neighbourhood  of  each  element. 


3.0  EXAMPLES 


3.1  Color  Segmentation 

We  digitized  the  color-transparency  photograph  of  a  canvas-back 
truck  with  3  different  color  filters  to  produce  the  color-separation 
images  shown  in  Fiqs.  l(a)-(c).  We  used  the  interactive  digital 
image-process inq  system  described  in  Ref.  13  with  Kodak  Wratten  types  25 
(red),  58  (green)  and  47  (blue)  color  filters  placed  individually 
between  the  vidicon  scanner  and  the  transparency  photograph.  The 
intensity  image,  i.e.  the  sum  of  the  red,  the  green  and  the  blue 
color-component  images,  is  shown  in  Fig.  1(d).  For  these,  and  for  all 
subsequent  photographs  given  here,  the  contrast  has  been  normalized  so 
that  the  highest  gray  level  in  the  image  is  displayed  as  full  white. 

Each  of  the  individual  color  images  was  divided  by  the  intensity 
image,  and  the  resulting  normalized  color-component  images  are  shown  in 
Fig.  2(a).  A  small  amount  of  smoothing,  performed  by  using  a 
2-dimensional  low-pass  filter  with  a  Gaussian  point-spread  function, 
produced  the  images  given  in  Fiq.  2(b).  Increasing  the  amount  of 
smoothing  yielded  the  3  images  shown  in  Fig.  2(c).  Such  smoothing 
reduces  the  small  variations  in  the  color  of  extended  homogeneous 
regions,  which  may  be  of  little  or  no  interest  for  the  present 
classification  purposes,  but  at  the  expense  of  blurring  the  edges 
between  differently  colored  regions. 
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First,  we  will  determine  which  of  the  color-ratio  Features  shown  in 
Fiq.  2  are  most  useful  for  dif ferent iat inq  the  olive-drab  colored  truck 
from  its  backqround  of  qreen  foliaqe  and  veqetation.  The  2  binary  masks 
shown  in  Fiq.  3  allow  us  to  individually  measure  the  color  contents  of 
the  tarqet  reqion  (Fiq.  3(a))  and  of  the  backqround  reqion  (Fiq.  3(b)). 
Figure  A  shows  the  color  content  of  the  complete  scene  (the  left  column 
of  A  ohotoqraphs)  of  the  tarqet  reqion  (the  center  column)  and  of  the 
backqround  reqion  (the  right  column).  The  color-content  information  is 
displayed  in  the  form  of  2-dimensional  distribution  functions  where  the 
gray  level  represents  the  probability.  A  high  probability,  which  means 
that  a  larqe  number  of  image  elements  have  that  particular  combination 
of  the  2  color  features,  appears  as  white,  whereas  a  low  probability  is 
displayed  as  black.  To  increase  the  dynamic  range  of  the 
distribution-function  displays,  the  gray  level  is  made  proportional  to 
the  square  root  of  the  probability.  To  visualize  the  3-dimensional 
color  space,  we  have  used  A  different  2-dimensional  distribution 
functions.  The  horizontal  and  vertical  axes  for  each  of  the  A  rows  are 
shown  to  the  left  of  the  photographs.  For  example,  for  the  top  row,  the 
horizontal  axis  represents  the  intensity  whereas  the  vertical  one 
represents  the  normalized  red  content. 

The  first  2  rows  of  images  given  in  Fig.  A  revpal  that  the  tarqet 
and  the  backqround  regions  are  poorly  separated  in  intensity.  The 
probability  for  both  regions  is  distributed  across  most  of  the 
horizontal  intensity  axis,  i.e.  both  regions  have  intensity  values 
within  the  name  range.  Setting  a  sinnle  intensity  threshold  will, 
therefore,  not  distinguish  the  2  regions  from  each  other.  The  third  row 
of  Fiq.  A  shows  that  the  2  regions  are  also  noorlv  separated  in  green 
content.  (In  the  other  hand,  from  rows  3  and  A,  we  see  that  the  tarqet 
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FIGURE  3  -  The  binary  masks  used  to  select  the  target  and  background 
regions  are  shown  in  (a)  and  (b)  after  multiplication  by  the 
intensity  image. 
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FIGURE  4  -  Color  content  of  the  scene.  Two-dimensional  distribution 
functions,  which  show  the  color  content  of  the  scene,  are 
displayed  in  the  form  of  gray  scale  images  where  gray  level 
is  proportional  to  the  square  root  of  the  probability.  The 
color  components  corresponding  to  the  horizontal  and  the 
vertical  axes  are  shown  to  the  left  of  each  row.  The  left 
photograph  in  each  row  was  obtained  by  using  the  complete 
image  area,  whereas  the  center  and  right  ones  were  obtained 
by  using  masks  to  select  the  target  and  the  background 
regions. 


UNCLASSIFIED 

14 


and  the  background  regions  are  well  separated  in  both  normalized  red  and 
blue  contents.  The  truck  target  has  a  higher  normalized  blue  content 
and  a  lower  normalized  red  content  than  the  background. 

Because  of  the  low  absorption  in  the  red  and  the  near  infrared  of 
chlorophyll  and  of  other  botanical  colorants  (Ref.  14),  the  reflectivity 
of  vegetation  increases  rapidly  from  approximately  10%  to  65%  at  700  nm. 
On  the  othnr  hand,  the  reflectivity  of  many  man-made  objects  (painted 
surfaces  etc.)  is  relatively  constant  to  beyond  1500  nm  (Ref.  15). 
These  properties  account  for  the  present  results  which  reveal  that  the 
background  has  a  higher  normalized  red  content  than  the  truck.  This  may 
be  useful  as  a  general  technique  for  enhancing  the  images  of  man-made 
objects  against  natural  backgrounds,  such  as  foliage  or  vegetation.  For 
example,  if  we  form  2  images  of  a  scene,  one  in  blue  light  and  the  other 
in  red  light,  and  divide  the  blue  image  by  the  red  one,  man-made  objects 
will  usually  appear  with  a  higher  gray  level  than  vegetation  (Ref.  7). 
In  some  cases,  it  may  be  possible  to  segment  such  objects  from  this 
background  by  setting  a  single  threshold  level  on  the  color-ratio 
feature . 

By  using  the  locations  of  the  tarqet  and  of  the  background  clusters 
in  the  normalized  color  space,  we  now  attempt  to  classify  the  individual 
image  elements  as  parts  of  either  target  or  background.  As  noted 
earlier,  smoothing  the  color  -component  images  can  improve  the 
classification  accuracy  at  the  expense  of  a  loss  in  the  spatial 
resolution  of  the  classified  regions.  Figure  5  illustrates  how  3 
different  amounts  of  smoothing  affect  the  distribution  of  the  normalized 
red  content  and  the  normalized  blue  content  as  measured  in  the  complete 
image,  the  target  and  the  background  regions.  Increasing  the  amount  of 
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(b)  Small  amount 
of  smoothing 


V 


(c)  Large  amount 
of  smoothing 


FIGURE  5  -  Improvement  in  color  classification  by  smoothing.  Increasing 
the  amount  of  spatial  smoothing  applied  to  the 

color-component  images  decreases  the  overlap  of  the  target 
and  the  background  clusters  in  the  3-dimensional  color  space, 
and  improves  the  ability  to  segment  the  2  regions. 
Photographs  of  the  distribution  of  the  normalized  red 
component  (horizontal  axis)  vs  the  normalized  blue  component 
(vertical  axis)  are  shown  for  no  smoothing  (a) ,  a  small 
amount  of  it  (b)  and  a  large  amount  of  it  (c) .  The  left 

photograph  in  each  row  was  obtained  by  using  the  complete 
image  area,  whereas  the  center  and  right  ones  were  obtained 
by  using  masks  to  select  the  target  and  the  background 
regions. 
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smoothing  reduces  the  overlap  between  the  target  and  the  background 
clusters.  A  certain  amount  of  overlap  remains;,  even  with  a  large  amount 
of  smoothing,  because  of  the  approximate  definition  of  the  target  and  of 
the  background  regions  by  the  masks  shown  in  Fig.  3  (e.g.  the  background 
mask  in  Fig.  3(b)  clearly  retains  some  of  the  target  region),  and 
because  of  the  blurring  at  the  target  edges  caused  by  the  smoothing. 
Note  that  a  large  amount  of  smoothing  reveals  structures  in  the  3 
distribution  functions  that  may  correspond  to  differently  colored  areas 
in  the  target  and  the  background  regions. 

We  can  see  from  Fig.  3  that  the  target  cluster  is  restricted  to  the 
upper  left  quadrant  of  the  space  spanned  by  the  normalized  red  component 
and  the  normalized  blue  component.  Figure  6  shows  the  result  of 
retaining  all  image  elements  that  have  normalized  color  contents  within 
these  limits.  Three  different  amounts  of  smoothing,  corresponding  to 
the  3  cases  used  in  Fig.  5,  produced  the  segmentation  results  given  in 
Figs.  6(b),  (c)  and  (d).  The  tradeoff  is  evident  between  a  good  spatial 
resolution,  as  seen  in  Fig.  6(b),  and  a  good  classification  accuracy,  as 
seen  in  Fig.  6(d). 

3.2  Texture  Segmentation 

We  aoplied  the  isotropic  and  the  linear  texture  algorithms  described 
in  Sect.  2.3  to  the  green  color  image  to  produce  the  binary  texture 
images  shown  in  the  left  column  of  Fig.  7.  The  center  and  right  columns 
in  this  figure  show  the  result  of  applying  a  moderate  amount  and  a  large 
amount  of  smoothing  to  the  2  binary  texture  images.  Figure  8  gives  the 
smoothed  texture  content  of  the  complete  image  (the  left  column),  of  the 
target  region  (the  center  column)  and  of  the  background  region  (the 
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(c)  Small  amount  of  smoothing  (d)  Large  amount  of  smoothing 


FIGURE  6  -  Segmentation  based  on  color.  The  truck  target  is  segmented 
from  the  original  scene  (a)  by  selecting  a  rectangular  volume 
which  contains  the  target  cluster  in  the  red-blue 
normal i zed-color  space.  When  no  smoothing  is  applied  to  the 
color-component  images  (b) ,  target  detail  is  sharp  but  there 
are  many  incorrectly  classified  points  in  both  the  target  and 
the  background  regions.  As  shown  in  (c)  and  (d) ,  more 
smoothing  increases  the  fraction  of  image  elements  that  are 
correctly  classified,  but  at  the  expense  of  a  loss  in  target 
spatial  resolution. 
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No  smoothing 


Moderate  amount  Large  amount 

of  Smoothing  of  smoothing 


FIGURE  7  -  Texture  components.  The  isotropic  texture  content  of  the 
scene  is  shown  in  (a) ,  and  the  linear  texture  content,  in 
(b) .  The  2  left  photographs  are  binary  images  obtained  by 
setting  an  element  to  "l"  if  the  3-  by  3-element  region  that 
surrounds  it  in  the  original  scene  possesses  that  texture 
attribute,  or  setting  it  to  ”0"  if  it  does  not.  A  moderate 
amount  of  smoothing  (center  image)  or  a  large  amount  of  it 
(right  image)  is  required  to  make  the  texture  images  suitable 
for  classification  purposes. 
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Complete  image 


Target  region 


Background  region 


(b)  Large  amount 
of  smoothing 


FIGURE  8  -  Texture  content  of  the  scene.  Two-dimensional  distribution 
functions,  which  show  the  texture  content  of  the  scene,  are 
displayed  in  the  form  of  gray  scale  images  where  the  gray 
level  is  proportional  to  the  square  root  of  the  probability. 
The  horizontal  ax is. cor responds  to  the  isotropic  texture  and 
the  vertical  one  to  the  linear  texture.  A  small  amount  of 
smoothing  was  used  to  produce  the  images  shown  in  (a) , 
whereas  a  large  amount  of  it  was  used  to  produce  those  shown 
in  (b) .  The  left  photograph  in  each  row  was  obtained  by 
using  the  complete  image  area,  whereas  the  center  and  right 
ones  were  obtained  by  using  masks  to  select  the  target  and 
the  background  regions. 
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right  column).  The  2-dimensional  distribution  functions  shown  in  this 
figure  were  produced  in  the  same  way  as  the  color  distribution  functions 
described  in  Sect.  3.1.  The  horizontal  axis  of  each  distribution 
function  corresponds  to  the  isotropic  texture  and  the  vertical  one  to 
the  linear  texture. 

In  comparing  the  center  and  right  columns  of  Fig.  8,  one  sees  that 
the  target  region  has  less  isotropic  texture,  but  more  linear  texture, 
than  the  background  region.  One  would  expect  that  the  truck,  which  is 
composed  of  relatively  smooth  surfaces,  will  have  less  isotropic  texture 
than  the  more  complex  background  of  foliage  and  vegetation.  On  the 
other  hand,  the  truck  image  contains  more  straight  edges  than  the 
background,  and  this  leads  to  a  higher  linear-texture  content. 

The  target  cluster,  therefore,  is  located  mainly  in  the  upper  left 
quadrant  of  the  isotropic  texture  -  linear  texture  distribution 
function.  Figures  9(b)  and  (c)  illustrate  the  target  segmentations  that 
result  when  we  retain  all  image  elements  having  textural  values  within 
these  limits.  The  2  results  were  obtained,  respectively,  by  applying  a 
moderate  and  a  large  amount  of  smoothing  to  the  texture  images  before 
classification.  Figure  9(d)  shows  the  result  if  one  uses  a  large  amount 
of  smoothing,  but  raises  the  horizontal-texture  threshold  and  lowers  the 
isotropic-texture  threshold.  This  reduces  the  number  of  background 
elements  that  are  incorrectly  classified  as  part  of  the  target,  but  at 
the  expense  of  increasing  the  number  of  target  elements  that  are  missed. 
It  is  evident  that  a  large  amount  of  smoothing  is  required  to  obtain  an 
acceptable  target  segmentation,  and  that  relatively  wide  classification 
limits  must  be  used  to  ensure  a  sufficiently  low  miss  rate.  The  Dresent 
textural  features  are  useful  for  locating  the  general  target  area,  but 
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they  cannot  give  significant  information  on  the  target  shape. 


3.3  Color-Texture  Segmentation 


Section  3.1  illustrated  that  color  ratios  can  be  effective  local 
features  to  use  for  image  classification.  If  random  fluctations  in  the 
color-ratio  features  (due  either  to  system  noise  or  to  variations  in  the 
intrinsic  spectral  reflectivities  of  the  surfaces)  are  sufficiently  low, 
then  little  or  no  spatial  averaging  may  be  reguired.  The  spatial 
resolution  of  the  regions  after  classification  may  be  close  to  that  of 
the  imaging  system.  However,  regions  which  have  the  same  spectral 
contents  but  which  belong  to  different  classes  cannot  be  distinguished, 
and  other  information,  such  as  texture,  must  be  used. 

Unless  a  surface  is  composed  of  closely  spaced  simple  repetitive 
patterns,  its  local  textural  properties  will  fluctuate.  In  Sect.  3.2, 
we  saw  that  it  may  be  necessary  to  use  a  considerable  amount  of  spatial 
smoothing  before  the  present  isotropic  and  linear-texture  features  are 
useful  for  region  classification.  The  spatial  resolution  of  a  texture 
classification  will,  in  general,  be  low  and  relatively  wide 
classification  limits  must  be  used  to  ensure  that  the  fluctuations  do 
not  result  in  too  high  a  miss  rate  (i.e.  a  high  false-alarm  rate  must  be 
tolerated).  In  the  present  class  of  imagery,  texture  clearly  provides 
less  useful  classification  information  than  do  color  ratios.  However, 
when  it  is  suitably  combined  with  color  content,  texture  can  supply 
additional  information  that  can  improve  classification  accuracy. 


1 
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Figure  10  shows  the  result  of  segmenting  the  truck  target  from  its 
background;  this  operation  is  based  on  the  truck  normalized  red  content, 
normalized  blue  content,  isotropic  and  linear  textures.  The 
segmentation  result  given  in  Fig.  10(b)  was  obtained  by  applying  a  large 
amount  of  smoothing  to  the  2  textural  features  and  a  small  amount  of  it 
to  the  2  color-ratio  features.  For  Fig.  10(c),  the  amount  of  textural 
smoothing  was  not  changed,  but  the  amount  of  color-ratio  smoothinq  was 
increased.  The  combined  use  of  the  color  ratio  and  the  textural 
features  has  preserved  the  high  spatial  resolution  of  the  color 
classif ications  shown  in  Fig.  6,  but  has  removed  regions  of  the 
background  which  were  incorrectly  classified  as  part  of  the  target 
because  they  had  the  same  color  content  as  the  target.  In  Fig.  10(b), 
note  the  sharply  defined  target  edges  which  follow  the  true  truck 
silhouette,  and  the  correct  classification,  as  belonging  tn  the 
background,  of  foliage  viewed  through  the  truck  windows. 


4.0  DISCUSSION 

This  document  was  intended  to  show  that,  in  pufff  sitv.ti'  rons,  the 
combined  use  of  appropriate  color  and  textural  lectures  is  suitable  for 
image  segmentation.  The  color  information  contributes  a  high  spatial 
resolution  to  the  segmentation,  whereas  the  textural  information  aids  in 
correctly  classifying  different  regions  that  may  have  similar  color 
properties.  We  do  not  claim  that  the  textural  measures,  the  color 
measures  nor  the  classification  approach  employed  here  are,  in  any 
sense,  optimum.  For  example,  nonlinear  spatial  smoothing,  performed 
either  before  or  after  the  classification,  may  yield  better  results  than 
the  present  linear  smoothinq.  Furthermore,  for  computational 
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(a)  Original 


(b)  Segmented  -  snail  amount  of  smoothing  (c)  Segmented  -  moderate  smoothing 


FIGURE  10  Segmentation  based  on  color  and  texture.  The  truck  target  is 
segmented  from  the  original  scene  (a)  by  selecting  an 
appropriate  volume  in  the  4-dimensional  space  spanned  by  3 
local  color  features  and  2  local  texture  features.  In  (b) ,  a 
small  amount  of  smoothing  was  used  before  each  individual 
picture  element  was  classified  as  belonging  to  the  target  or 
the  background.  This  produces  a  sharply  defined  target 
region  (note  the  accuracy  with  which  the  upper  and  lower 
edges  of  the  segmented  region  follows  the  true  target 
contours,  and  the  correct  classification,  as  belonging  to  the 
background  region,  of  foliage  that  is  viewed  through  the 
truck  windows);  however,  several  small  regions  of  the  image 
are  incorrectly  classified.  Increasing  the  amount  of 
snoothing  (c)  reduces  the  miss-classification  rate,  but  at 
the  expense  of  a  blurring  of  target  detail. 
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simplicity,  we  selected  rectanqular  regions  in  the  color-texture  space. 
As  is  evident  in  the  experimental  distribution  functions  given  in  Sect. 
3.0,  more  image  elements  would  have  been  correctly  classified  had  we 
used  arbitrarily  shaped  (curved)  segmentation  limits. 

We  obtained  the  present  results  by  observing  the  locations  of  the 
target  and  the  background  clusters  in  the  color-texture  space  and  then 
choosing  appropriate  limits  to  seqment  the  target.  These  limits  apply 
only  to  the  present  case,  and  may  not  be  appropriate  for  other  images. 
In  a  practical  imaqe  classification  system,  it  is  usually  necessary  to 
adaptively  adjust  the  classification  limits  to  suit  the  particular  input 
imagery.  Systems  intended,  for  example,  to  classify  areas  in  satellite 
imagery  or  to  interpret  printed  characters  have  successfully 
accomplished  this  by  adaptive  training  (Ref.  16),  i.e.  the  limits  are 
established  by  providing  the  system  with  preclassified  images.  It  has 
not  been  established  to  which  extent  such  an  adaptive  training  approach 
would  succeed  in  correctly  classifying  the  images  of  3-dimensional 
objects  which  may  appear  with  different  orientations,  under  different 
illumination  conditions,  or  which  may  be  partially  obscured.  However, 
it  is  clear  that  features,  such  as  color  ratios  and  texture,  which  are 
relatively  independent  of  such  viewing  conditions,  will  be  more  suitable 
than  those  which  are  dependent. 

In  some  situations,  it  may  be  possible  to  choose  features  which, 
when  combined  with  a  priori  knowledge,  may  allow  us  to  achieve  correct 
classification  with  little  or  no  adaptive  training.  For  example,  the 
use  of  a  Fixed  threshold  on  the  blue-to-red  color  ratio  described  in 
Sect.  3.1,  may  permit  the  segmentation  of  a  wide  variety  of  man-made 
targets  from  a  background  of  vegetation. 
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We  performed  the  present  study  by  simulation  on  an  interactive 
digital  image-processing  system,  but  all  processing  described  here  is 
currently  feasible  at  standard  TV-video  rates.  For  example,  real-time 
video  digitizers  are  available  as  is  the  digital  video  circuitry  to 
perform  division  and  3-  by  3-element  spatial  convolution.  Hardware  to 
carry  out  multifeature  classification  at  30  TV  frames/s  is  under 
development  by  at  least  one  private  industry.  New  approaches,  such  as 
the  use  of  adaptive  tree-based  logical  processing  (Refs.  16  and  17),  may 
permit  high-efficiency  implementations  of  complex  imaqe-segmentation 
algorithms . 


5.0  CONCLUSION 

We  considered  the  problem  of  classifying  the  elements  of  a  digital 
image  formed  by  the  reflection  of  radiation  from  a  scene  into  different 
categories,  such  as  target  or  background.  Only  local  features,  which 
depend  either  on  the  properties  of  the  image  element  or  on  those  of 
elements  within  a  small  neighbourhood  of  it,  were  employed;  macroscopic 
properties  of  the  regions,  such  as  shape,  aspect  ratio,  context  etc., 
were  not  used.  We  should  choose  local  features  which  measure  properties 
of  the  objects  in  the  scene,  and  that  are  independent  of  viewinq 
conditions,  such  as  surface  orientation  and  illumination.  Color-ratio 
and  textural  features  can  satisfy  these  requirements,  and  we  examined 
their  combined  use  for  image  classification. 

If  the  objects  in  the  scene  have  constant  spectral  reflectivities, 
and  if  the  noise  level  introduced  during  the  image  formation  and 
recording  processes  is  sufficiently  low,  the  spatial  resolution  of  a 
color-ratio  classification  can  be  equal  to  that  of  the  imaging  system 
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itself.  In  such  cases,  point  targets,  as  well  as  extended  targets,  can 
be  classified  with  color-ratio  features. 

Visual  texture  is  only  useful  for  classifying  extended  regions  that 
are  well  resolved  by  the  imaging  system  (e.g.,  that  occupy  10  elements 
or  more  across  their  minimum  dimension).  In  practice,  the  textural 
properties  of  most  homogeneous  surfaces  undergo  relatively  large 
fluctuations,  and  a  significant  amount  of  spatial  smoothing  is  reguired, 
either  before  or  after  the  texture  classification  is  performed.  The 
spatial  resolution  of  a  texture  classification  may,  therefore,  be  low. 
Nevertheless,  texture  may  be  a  useful  feature  for  distinguishing  between 
extended  regions,  in  a  scene,  that  cannot  be  separated  easily  by  other 
means.  For  example,  it  may  be  relatively  easy  to  camouflage  a  target  so 
that  it  resembles  the  average  color  of  its  background  (a  land  target  may 
be  painted  green  to  imitate  vegetation,  a  ship  may  be  painted  blue-gray 
to  match  water,  etc.),  but  it  is  more  difficult  to  match  their  textural 
properties.  Note  that  the  matching  of  visual  color  or  texture  for 
camouflaging  purposes  does  not  guarantee  that  appropriate  processing 
algorithms  could  not  distinguish  the  target  from  its  background. 

The  combined  use  of  color  and  texture  information  may  be  useful  for 
automatically  detecting  or  tracking  visible-light  images  of  military 
targets  against  complex  backgrounds.  In  practice,  the  color  and  the 
textural  properties  of  a  surface  are  usually  unrelated  to  one  another, 
and  each  forms  an  independent,  but  complementary,  basis  for  region 
classification.  Such  automatic  processing  should  be  optimized  for  a 
particular  imaging  situation  by  using  all  the  a  priori  information 
available.  Most  color  and  texture  algorithms  are  computationally 
simple,  and  relatively  easy  to  implement  in  real-time  hardware.  Further 
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experiments  with  other  types  of  military  imagery  are  required  to 
determine  the  range  of  possible  applications  of  color-texture 
segmentation. 
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